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, Abstract. In this paper we study the branching law for the 

restriction from SU{n,m) to SO{n,m) of the minimal represen- 
tation in the analytic continuation of the scalar holomorphic dis- 
crete series. We identify the the group decomposition with the 
spectral decomposition of the action of the Casimir operator on 
' the subspace of S{0{n) x 0(m) )-invariants. The Plancherel mea- 

"j^ , sure of the decomposition defines an L^-space of functions, for 

which certain continuous dual Hahn polynomials furnish an or- 
thonormal basis. It turns out that the measure has point masses 
precisely when n — m > 2. Under these conditions we construct an 
irreducible representation of SO{n, m), identify it with a parabol- 
ically induced representation, and construct a unitary embedding 
] into the representation space for the minimal representation of 

■ SU{n,m). 

m 

'. 1. Introduction 



One of the most important problems in harmonic analysis and in 
representation theory is that of decomposing group representations 
into irreducible ones. When the given representation arises as the 
restriction of an irreducible representation of a bigger group, the de- 
composition is referred to as a branching law. One of the most famous 
^ ■ examples of this is the Clebsch-Gordan decomposition for the restric- 

tion of the tensor product of two irreducible 5'f/(2)-representations 
(which is a representation of SU (2) x SU (2)) to the diagonal subgroup. 
For an introduction to the general theory for compact connected Lie 
groups, we refer to [TI] . 

Since the work by Howe ([7]) and by Kashiwara-Vergne ([9]), the 
study of branching rules for singular and minimal representations on 
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spaces of holomorphic functions on bounded symmetric domains has 
been an active area of research. In [H|, Jakobsen and Vergne studied 
the restriction to the diagonal subgroup of two holomorphic represen- 
tations. More recently, Peng and Zhang ([22|) studied the correspond- 
ing decomposition for the tensor product of arbitrary (projective) rep- 
resentations in the analytic continuation of the scalar holomorphic 
discrete series. Zhang also studied the restriction to the diagonal of a 
minimal representation in this family tensored with its own anti-linear 
dual ([33]). 

The restriction of the representations given by the analytic con- 
tinuation of the scalar holomorphic discrete series to symmetric sub- 
groups (fixed point groups for involutions) has been studied recently 
by Neretin ([H], ^7\), Davidson, Olafsson, and Zhang ([2]), Zhang 
([32], [34]), van Dijk and Pevzner ([30]) and by the author ([28]). 

All the above mentioned decompositions have the common feature 
that they are multiplicity free. This general result follows from a 
recent theorem by Kobayashi ([I3]), where some geometric conditions 
are given for the action of a Lie group as isometric automorphisms 
of a Hermitian holomorphic vector bundle over a connected complex 
manifold to guarantee the multiplicity-freeness in the decomposition of 
any Hilbert space of holomorphic sections of the bundle. The action 
of a symmetric subgroup on the trivial line bundle over a bounded 
symmetric domain then satisfies these conditions (cf. [T4]). 

In this paper we study the branching rule for the restriction from 
G := SU{n,m,) to H := SO{n,m) of the minimal representation in 
the analytic continuation of the scalar holomorphic discrete series. 
We consider the the subspace of L := S{0{n) x 0(m) )-invariants and 
study the spectral decomposition for the action of Casimir element of 
the Lie algebra of H. The diagonalisation gives a unitary isomorphism 
between the subspace of L-invariants and an L^-space with a Hilbert 
basis given by certain continuous dual Hahn polynomials. The main 
theorem is Theorem [9l where the decomposition on the group level is 
identified with this spectral decomposition. The Plancherel measure 
turns out to have point masses precisely when n — m > 2. The second 
half of the paper is devoted to the realisation of the representation 
associated with one of these points and the unitary embedding into 
the representation space for the minimal representation. The main 
theorem of the second half is Theorem [2TJ 
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The paper is organised as follows. In Section 2 we begin with some 
preliminaries on the structure of the Lie algebra 0, the group action, 
and the minimal representation. In Section 3 we construct an or- 
thonormal basis for the subspace of L-invariants. In Section 4 we 
compute the action of the Casimir elements on the L-invariants and 
find its diagonalisation. We also state the branching theorem. In Sec- 
tion 5 we construct an irreducible representation of the group H (for 
n — m > 2, i.e., when point masses occur in the Plancherel measure), 
identify it with a parabolically induced representation, and finally we 
construct a unitary embedding that realises one of the discrete points 
in the spectrum. 

Acknowledgement: The author would like to thank his advisor 
Professor Genkai Zhang for support and for many valuable suggestions 
during the preparation of this paper. 



2. Preliminaries 
Let ^ be the bounded symmetric domain of type /mn(^ ^ , i-e., 

(1) ^-.^{ze MnmiC)\In - ZZ* > 0} . 

Here M„m(C) denotes the complex vector space of n x m matrices. 
We let G be the group SU{n,m), i.e., the group of all complex {n + 
m) X (n + m) matrices of determinant one preserving the sesquilinear 
form (-, ■)n,m on C"""*"™ given by 

{2){U, V)n,m = UiVi H h UnVn " Un+lVn+1 Un+mVn+m- 

The group G acts holomorphically on ^ by 

(3) g{z)^{Az + B){Gz + Dr\ 

\i g — ^^isa block matrix determined by the size of A being 

n X n. The isotropy group of the origin is 
K := S(U(n) x U{m)) 

= {(^0 ^ ) 1^ e t/(n),D e ?7(m),det(A)det(D) = l| , 
and hence 

(4) ^ ^ G/K. 
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2.1. Harish-Chandra decomposition. Let ^ denote the Cartan in- 
volution g ^ (g*)^^ on G. We use the same letter to denote its differ- 
ential 6* : 5 — > at the identity. Here, we have identified Te{G) with 
0. Let 



(5) 



g = « ©P 



be the decomposition into the ±1 eigenspaces of 9 respectively. In 
terms of matrices, 



(6) fi 

(7) P 



A 
D 



\A* = -A, D* = -D, tr{A) + tr{D) = 





B* 



B 




where the size A is n x n. 

The Lie algebra Q has a compact Cartan subalgebra t C fi, where 

( ( tsi • • • • • • \ 



(8) t: 















iti 








3 "1 







I V • • • • • • itm J 

Its complexification, (the set of complex diagonal traceless matri- 
ces), is a Cartan subalgebra of the complexification ^ — si{n-\-m, C), 
where 



(9) 



We let Eij denote the matrix with 1 at the entry corresponding to the 
ith row and the jth column and zeros elsewhere. By E*^ we mean the 
dual linear functional, i.e., E*j{z) = Zij for z e M„^(C). Moreover, 
we define an ordered basis {Fj} for t*^ by 



(10) 



E*, - E*, 



33 J 



,n -\- m — 1, 



The root system, A(0^,t'^) is given by 
(11) A(0^, t^) = {El -Em<i,j<n + m,iy^ j}. 
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We denote the root E*^ — E*j by aij. We define a system of positive 
roots, A+, by the ordering (fTOll . Then 

(12) A+ = {«,,|j>2}, 

and we let A~ denote the complement so that A = A+ U A~. For 
a root, a, we let stand for the corresponding root space. Then 
= CEij. For a root space, g", we either have g" C or g° C p*^. 
In the first case, we call the corresponding root compact, and in the 
second case we call it non-compact. We denote the sets of compact 
and non-compact roots by A^ and Ap respectively. Finally, we let Ap 
and A~ denote the set of non-compact positive roots and the set of 
non-compact negative roots respectively. We set 

(13) P+ = Efl"' 

(14) p- = Es"- 



These subspace are abelian Lie subalgebras of p . Moreover, the rela- 
tions 

(15) [«^p1Cp+,[!C,p-]Cp-,[p+,p-]C!C 

hold. We let K^,P~^, and P~ denote the connected Lie subgroups 
of the complexification of G, G^ , with Lie algebras fi'^,p"'", and p~ 
respectively. The exponential mapping exp : p^ — P^ is a diffeomor- 
phic isomorphism of abelian groups. As subspaces of the Lie algebra 
= sl{n + m) we have the matrix realisations 

(16) ^ {(o 

(17) P" = (f ° n 1 IzeM^rXQ 



The Lie algebra g''' can be decomposed as 

(18) g^ = p+©«^©p". 
On a group level, the multiplication map 

(19) P^ xK^ xP- G^, (p, k, q) ^ pkq 
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is injective, holomorphic and regular with open image containing GP^ . 
In fact, identifying the domain & with the subset 

Q Q ^ I2; e ^1 C p+ and letting 



, , /„ 
there is an inclusion 

(20) GP+ C VLK^P-. 

For g E G, we let {g)+,{g)o, and (g)- denote its P~^,K^, and P~ 
factors respectively. The action of g on ^ defined by 

(21) ^(2;) = log((^exp2;)+) 
then coincides with the action ([3]). In fact, iov g = ( ^ ^ 



Harish-Chandra factorisation is given by 
(22) 



A B 
C D 

In BD-^ \(^- BD-^C \ f In 
/™ j V D [ D-'C In, 



For g as above, and exp z = j ^ , 

(23) ^7expz= ^^^^ J 



and hence 



(24) (jexp.U^I + + 



-1 



by ((221). 

We also use the Harish-Chandra decomposition to describe the dif- 
ferentials dg{z) for group elements g at points z. We identify all 
tangent spaces T^(^) with p+(= M„^(C)). Then dg{z) : p+ ^ p+ is 
given by the mapping 

(25) dg{z) = M{{ge^vz)o)\p+ 
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(cf. [26]). In the explicit terms given by ( l22ll . this mapping is given 

by 

dg{z)Y = {A- {Az + B){Cz + D)-^C)YD~\ Y G M„„(C). 

2.2. Strongly orthogonal roots. We recall that two roots, a and 
/?, are strongly orthogonal if neither a + nor a — /3 is a root. We 
define a maximal set of strongly orthogonal noncompact roots, F, in- 
ductively by choosing 7^+1 as the smallest noncompact root strongly 
orthogonal to each of the members {71, . . . , 7^} already chosen. When 
the ordering of the roots in given as in ( fTOl ). we get 

(26) r = {71, . . . , 7^}, 7j- = E*j - 

We now let E^. denote the elementary matrix that spans the root 
space . Then the real vector space 

n 

(27) a:=Y,nEr,-OE,^) 

is a maximal abelian subspace of p. We set 

(28) Ej := E^^ - 9E^^. 

2.3. Shilov boundary. Let O(^) denote the set of holomorphic func- 
tions on ^, and let O(^) denote the subset consisting of those which 
have continuous extensions to the boundary. The Shilov boundary of 
^ is the set 

S = {ZE V\Im-Z*Z = 0}. 

It has the property that 

(29) sup\f{z)\= snplf {z)\JeOm, 

and it is minimal with respect to this property, i.e., no proper subset 
of S has the property. The set S can also be described as the set of all 
rank m partial isometrics from C"* to C". The group K = U{n) x U{m) 
acts transitively on S by 

{9,h){z) = gzh~^. 
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To find the isotropy group of the fixed element Zq :- 
{g, h) & U{n) X U (m) and write g in the form 



, let 



9 = 



A B 



C D 

where A is of size m x m. Then 

A B\ f h-^ 



gzoh ^ 



C D 







Ah-^ 
Ch-^ 



So, the equality gzohr^ = zq holds if and only if A = h and C = 0. 
Since g is unitary, the last condition implies that also B = and hence 
the isotropy group is 



Ko := {U{n) x U{m)) 



{g,h) eU{n)x U{m)\g 



h 
D 



Thus we have the description 

S = K/Ko = {U{n) X U{m))/U{n - m) x U{m) 

of the Shilov boundary homogeneous space. 

In the sequel, we will often be concerned with the submanifold 5a 
of where 



(30) cSa := < 



/6 




V 



\ 



> . 



/ 



Also, we let diag(^) denote the m x m-matrix 
identity 

(31) = f dia^K) A / diagK) 



••• 
: ■■. 
••• U 







The 



identifies the matrices in the submanifold 5a with certain cosets in 
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2.4. The real form . Consider the mapping r : ^ — > ^ defined 

by 

(32) t{z)^z, 

where the conjugation is entrywise. It is an antiholomorphic involutive 
diffeomorphism of Ql. We let 2J denote the set of fixed points of r, 
i.e., 

(33) = e Q)\t{z) = z\. 

Moreover, r defines an involution, which we also denote by r, of G 
given by 

(34) r{g)^TgT-\ 

We let H denote the set of fixed points, i.e., 

(35) E^G^^{g^ G\T{g) = g) . 

Clearly, H = SO{n,'m), i.e., the elements in G with real entries. The 
group H acts transitively on ^ , and the isotropy group of in if is 
L:^Hr\K. Hence 

(36) 3^ ^ H/L. 

2.5. Minimal representation We recall that the Bergman ker- 
nel of S> is given by 

(37) K{z, w) = det(4 - ^w*)-("+"*). 
It has the transformation property 

-1 



(38) K(gz,gw)^ Jg{z)-^K{z,w)Jg{w) , 

where Jg{z) denotes the complex Jacobian of g at z. We let h{z,w) 
denote the function 

(39) h{z, w) = det(4 - zw*). 
Then, for real i/, the kernel 

(40) M-,-)-'^ 

is positive definite if and only if v belongs to the Wallach set, W. 
Here, 

(41) W-{0,l,...,m-l}|J(m-l,oo) 
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(cf. [3]). The kernel h{-,-) ^ satisfies the transformation rule 

(42) Kgz.gw)-^ = Jg{z)-^h{z,w)-''l^'^ . 

For u G W, we denote the Hilbert space defined by the kernel ■)^^ 
by J^y. A projective representation, iiy, of G is defined on by 

(43) TxMf{z) = JM^)'^f{9-'z). 

We will be concerned with the so called minimal representation, i.e, 
with the representation tti on the space 

3. The L-invariants 

For any G W, let 

= 

fc: = -(fcl7lH hfcm7m) 

be the decomposition into i^-types. Here F = {71,..., 7^} is the 
maximal strongly orthogonal set in Ap with ordering 71 < ■ ■ ■ < 7m 
defined in the previous section, and 

(44) ki>--->k^,hen, 

and V- is a representation space for the i^'-representation of highest 
weight that is realised inside the space of homogeneous polynomials of 
degree = fci + ■ ■ ■ + fc^ on p"*". When = 1, the weights occurring 
in this sum are all of the form 

(45) k = -k^i 
(cf. [3]). Taking L-invariants, we have 

k 

The data {K, L, r) defines a Riemannian symmetric pair, and hence 
{V-)^ is at most one dimensional by the Cartan-Helgason theorem (cf. 
[6], Ch. IV, Lemma 3.6.). 

We recall the compact Cartan subalgebra t C 6 in (JH]). We let t 
denote the Cartan subalgebra of u{n) © u(m) consisting of all diag- 
onal imaginary matrices, i.e., matrices of the form ([8]) but without 
the requirement that the trace be zero. Then we have an orthogonal 
decomposition 

(46) t = t © 
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given by the Killing form. 

Any linear functional / G t* extends uniquely to a functional on t 
which annihilates the orthogonal complement t^. We will denote these 
extensions by the same letter /. Therefore, any dominant integral 
weight on t parametrises an irreducible representation of u{n) © u(m) 
in which t"*" acts trivially. When X = k = k'ji is a i^-type occurring 
in Jti, we denote the underlying representation space for u{n) ©u(m) 
by V^. Moreover, the Cartan subalgebra t is the sum 

t = tl © t2 

of the corresponding subalgebras of u(n) and u(m) respectively. The 
restrictions of A to ti and t2 respectively define integral weights, hence 
they parametrise irreducible representations of the Lie algebras u(n) 
and u(m) respectively. We denote the corresponding representation 
spaces by and In what follows, A will always denote the 

extension to u(n) © u(m) of a weight of the form k in (I45D . We will 
use the explicit realisations 

(47) =0 C", 

where the right hand side denotes the symmetric tensor product de- 
fined as a quotient of the fc-fold tensor product of C^. In the following, 
for a multiindex a = (ai, ■ ■ ■ , a„) G N", we let 

(48) |a| := ai + h 

(49) a\ := ai\---aj. 

For any choice of orthonormal basis {ci, . . . , e„} for C", the set 

(50) {e° :=e?^---e^"||a| = fc} 

k 

furnishes a basis for C". We fix an i^-invariant inner product, || ■ 
on C" by the normalisation 

(51) l|e?l|^ = A;!. 

Observe that we have suppressed both the indices k and n here. For 
n fixed, the norm in fact equals the restriction of the norm defined 
on all polynomial functions on C" (we use the natural identification 
^ 2;" of symmetric tensor power with polynomial functions) 

(52) {P,q)k:=p{d){q*m, 



This is often called the Fock-Fischer inner product (cf. [3J). 
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where p{d) is the differential operator defined by substituting ^ for 
Cj in p, and for q — Oq^;", q* is defined as 

(53) {^aazn*:=j2^^"- 

a a 

The suppressing of the index n will not cause any confusion in what 
follows. Finally, on the dual space we have the corresponding basis 

(54) {{er--=ieir---{e:nH = k}, 

where {e^, . . . , e*} is the dual basis to {ci, . . . , e,„} with respect to the 
standard inner product on C^. We also let || • denote the /T-invariant 
norm on normalised by 

(55) Wieim-kl 

Lemma 1. For any choice of orthonormal basis {ei, . . . , e„i} for C" 
and extension {ei, . . . , e^, e^+i, • • • , 6^} to an orthonormal basis for 
C", the vector 

\a\=k 

where fa = -(^tj2 and f* = (§yi^, is Ko-invariant. 

Proof. We recall the identification of the isotropic subgroup of the 
fixed element Zq with U{n — m) x U (m) . From this it is clear that 
it suffices to prove that the vector ix &Vm^ i^mY ^ ® (K^)* is 
invariant under the restriction of the representation of U (m) x U (m) 
to the diagonal subgroup. 

The vector space ® (V^)* is naturally isomorphic to End(V^), 
the isomorphism being given by {u<S>v*){y) — v*{y)u. Then, if y G 
is the linear combination y — YliB^pfp^ 



a o.,P 
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i.e., i\ corresponds to the identity operator. Moreover, for the action 
of u(m) on the tensor product (g) (V^)*, we have 

X( u<S>v*){y) — {Xu <^ v*){y) + {u ^ Xv*){y) 

= v{y)Xu+{Xv*){y)u 

— v{y)Xu + {y, Xv)u 

= v{y)Xu- {Xy,v)u 

= [X,u^v*]{y), 

where X e u{m),u e V^,v* e (V^)*, i.e., the action as derivations 
of the tensor product corresponds to the commutator action on the 
endomorphisms. In particular, Xix = for all X in u(m). This proves 
the lemma. □ 

Since the vectors in the representation space are holomorphic 
polynomials, they are determined by their restrictions to the Shilov 
boundary S. 

In the sequel, we use the Fock inner product to define an antilinear 
identification of with {V^)* by 

v^v*, v*{w) — {w,v)jr, weV^. 

We let (•, •) denote the inner product on the tensor product V^<S>{V^)* 
induced by the Fock inner products on the factors. 

Proposition 2. The operator Tx : ® {V^)* defined by 

Tx{u®v*){z) - {{g,h)ix,u^v*), 

where z = {g,h)Ko E S, is a C-antilinear isomorphism of U{n) x 
U (m) -representations. 

Proof. We first observe that the left hand side is well defined as a 
function of z by the invariance of ix- 

The root system A(u(n) © u(m), t) is the union of the root systems 
A(u(n), ti) and A(u(m), t2). Fix choices of positive roots A+(u(n), ti), 
and A+(u(m), respectively. We define a system of positive roots in 
A(u(n) © u(m), t) by 

A+(u(n) © u(m), t) := A+(u(n), ti) U A+(u(m), ta). 

Let Ma e be a lowest weight-vector, and G be a highest 
weight-vector. Then ux ® ^a is a lowest weight-vector in ® (Kn)*- 
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For H = {Hi, H2) G ii © t2 we have 

= ^((exp tHig, exp tH2h)Lx, ux ® vl)t=o 

= ^ {{{g, h)ix, (exp -tHi, exp -tH2){ux ® fI)))t=o 
= {{g,h)LxA{-Hi)ux®vl) 

+ {{g,h)ix,ux® Xi-H2)vl) 
= X{H)Tx{ux®vl){z). 

Thus Tx{ux ® vl) is a vector of weight A. 

Any root vector in u(n) © u(m) lies in either of the components. 
Take therefore a positive root vector E + iF E u(n)'^. Then 



{E + zF,0){Tx{ux®vl)){z) 
= ^ (((expt^^,/i)iA,MA ©^^I))t=o 

i{iexptFg,h)Lx,Ux®vl))^^^ 

= {{g,h)Lx,{-{E-iF)ux®vl)) 
= 0, 



since E — iF is dt negative root vector. Similarly one shows that the 
positive root vectors in u(m) annihilate Txiux ® v'l). The function 
Tx{ux®vD on the Shilov boundary naturally extends to a holomorphic 
polynomial on & which belongs to J^. Hence Txiux © v^) can be 
written as finite sum of highest weight-vectors from the i^'-types of 
Ml. But it is a vector of weight A, and so by the multiplicity- freeness 
of the i^-type decomposition, Tx{ux © ^a) is a highest weight-vector 



Lemma 3. The space {V^)^ is nonzero if and only if \ = —2kji for 
A; e N. In this case, it is one-dimensional with a basis vector ipk, where 



in V\ 



□ 




m=k 

where z^ is the matrix defined in (l30ll . 
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Proof. We use the isomorphism from the proposition above. Then the 
first statement is obvious, since for any A = — J7i, the representation 
space is isomorphic to the space of all polynomials of homogeneous 
degree j on C", and the corresponding statement holds for V^. Assume 
therefore that A = — 2/c7i. 

Clearly, the vector {ef -\ h e^)*^ ((e*)^ H h (e^J^)'' is an 

L-invariant vector in (V^)*. We compute its image under Tx 
when restricted to the matrices in Sa- 

= ((^^, (e? + • • • + elf ® {{e{f + • • • + (e;)^)'^) 

= (E ®r.Ml + --- + elf ® {{elf + . • • + (e;)^)^) 

a 

= E^'^^/- (e? + • • • + eif)u:, mf +■■■ + {eiff). 

a 

Since the symmetric tensor (e^ + - ■ ■+elf has the monomial expansion 
iel + --- + elf=Y.(l)e^^, 

m=k ^ ^ 

we get the equality 

Tx{{el + --- + elf®{{elf + --- + {elff){z,)=J2(3)\me^. 

m=k ^ ^ 

□ 

Theorem 4. The polynomials ipk of degree 2k, for k E N, given by 

constitute an orthonormal basis for the subspace, M'f , of L-invariants. 

Proof. The only thing that is left to prove is the normalisation part of 
the statement, i.e., we need to compute the norms of the polynomials 

i>k- 

Using the antilinear isomorphism Tx, we can introduce an inner 
product 



{■,-)'x:={Tx';Tx'-), 
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where the the right hand side denotes the conjugate of the inner prod- 
uct on the tensor product induced by the Fock inner products on the 
factors, on l^^. By Schur's lemma, the equality 

II ■ 11^ = C'aII ■ IIa 

holds on V'^ for some complex constant C\. To compute this constant, 
we compare the norms of the lowest weight- vector ux<S)vl and the high- 
est weigh-vector Tx{ux ^v^) in their respective representation spaces. 
Let {ei, . . . , Cm} and {ei, . . . , e^} denote the standard orthonormal 
bases for C" and C™ respectively. Then ux®vl = ef' ® (ef'^)*, and 

||ef ® (ef)*|| = (2A;)!. 
Moreover, the normalised lowest weight-vector ^ .^.s, maps to 



(2fc)! 



2fc ^ („*\'2k^ 

2k 



where 

where pu is the highest weight vector given by pn{z) = zf^. Since 
= ^/{2ky., we see that Cx = a/(2A;)!. 

The norm of (e^H ^e^)''® (((el;)^H \-{^m)^) is straightforward 

to compute. In fact, 

2\fc||2 ll//„*\2 I I /*\2\kll2 /..xo/^\ /"n 



ll(ef + ■ ■ ■ + eirr^ WiielY + ■ ■ ■ + (61)^11'^ = (kl, ^ ^ V2 
Finally, we have the equality 

(57) ll'll'^ (2^"'"^ 

(cf. [3]) relating the ^-norm to the Fock-Fischer norm on the i^'-type 
2k = —2^71, and this ends the proof. □ 

4. The action of the Casimir element on the L-invariants 

We consider the representation of the universal enveloping algebra 
U{[)^) defined for all X e () by 

(58) /h^^7ri(exptX)/|i=o, 



BRANCHING LAWS FOR MINIMAL HOLOMORPHIC REPRESENTATIONS 17 



for / in the dense subspace, of analytic vectors, and extended 

to a homomorphism U{1)'^) — ^ End(.^°°). We will denote this repre- 
sentation too by TTi. We recall that the Casimir element, C G f/(f)'^) is 
given by 

(59) C = Xf + --- + X^-Y,^ Yg^, 

where {Xj, i = 1, . . . , dimq} and {Yi, i = 1, . . . , diml} are any orthog- 
onal bases for q and I respectively with respect to the Killing form, 
B{-, ■), on f) such that 

B{Xi,Xi) = 1, z = 1, . . . ,dimq, 
B{Yi,Yi) = -1, 2 = 1,..., diml. 

Consider now the left regular representation, /, of H on C°°{H/L), 
i.e., l{h)f{x) = f{h-^x). We define an operator Ri : J^i C°°{H/L) 
by 

(60) RJ{x) ■=h{x,x)-^/^f{x). 

This is the generalised Segal-Bargmann transform due to Olafsson and 
0rsted (cf. [21]). A nice introduction to this transform in a more gen- 
eral context can also be found in Olafssons overview paper [20] • The 
following lemma is an immediate consequence of the transformation 
rule (1421). 

Lemma 5. The operator Ri : .W\ C^i^H/ L) is H-equivariant. 

Moreover, the Casimir element acts on C°°{H/L) as the Laplace- 
Beltrami operator, £, for the symmetric space H/L. We recall the 
"polar coordinate map" (cf. [5], Ch.IX) 

(61) (p : L/M X ^ {H/L)\ 

{IM, a) ^^ laL 

Here {H/L)' := H'/L, where H' is the set of regular elements in H, 
and = exp a"*" , where 

(62) a+ = {tiEi + ■■■ + tmE^mlU > 0, i = 1, . . . , m}. 

The map </> is a diffeomorphism onto an open dense set in H/L. Hence, 
any / G C°°{H/L)^ is uniquely determined by its restriction to the 
submanifold A"*" ■ = '?/'({eM} x A"*"). In fact, the restriction mapping 
/ ^ /U+ o defines an isomorphism between the spaces C°°{H/L)^ 
and C'°o(A+-0)^^(")/^^("). The space C^{H/Lf is invariant under the 
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Laplace-Beltrami operator. Recall that the radial part of the Laplace- 
Beltrami operator is a differential operator, A£, on the submanifold 
A'^ ■ with the property that the diagram 

C°^{H/L) C°°{H/L) 



C°°(A+-0) C°^(A+-0) 

where the vertical arrows denote the restriction map, commutes. 

Moreover, the functions in Jifi are determined by their restrictions 
to the real submanifold H/L, and the L-invariant functions are de- 
termined by their restrictions to A'^ ■ 0. By Lemma [5] and the above 
discussion, we have the following commuting diagram. 



C7-(A+-0) JlLJ^ C^{A+-0) 

where, again, the vertical arrows denote the restriction maps. 

In what follows, we will compute the action of the operator R^^ACRi 
on the subspace 

The radial part of the Laplace-Beltrami operator of H/L is given 
by (cf.d, Ch. II, Prop. 3.9) 



j=l -J Tn>i>j>l ■' 
™ Q 

+ (n-m)^cothtj — . 

The coordinates are related to the Euclidean coordinates Xi by 
tanhtj, i.e., 

(63) = {(xi,...,a;„)|0 < xi < < ■ ■ ■ < x„ < 1} 
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In the coordinates Xi, the operator AR^ ^ACRi := has the expres- 
sion 

i=l ^ * J / 

+ ^(2a:?-2x,(l-x,^)A'j 
1=1 ^ *^ 



m / 

+ (n — m) ( - 
i=i ^ 

+2 Y. (-1 + 



d 1 a 
-1 - V 



— V OXi OXj 

m>i>j>l ^ t J \ ^ J 

The following lemma is proved by a straightforward calculation. A 
proof for a similar decomposition can be found in |33| . 



Lemma 6. The operator AR^^ ACRi can be written as a sum of three 
operators, C^,Co and that lower, keep and, respectively, raise the 
degrees of the polynomials ipk ■ In fact, 

f d'^ n — m d \ \ ^ I f d d 
^- = / ^ TT^ Tr~ +2 > , — I Xi- X,- 



dxf Xi dxi J ^ xf — x'j \ ' dxi dxi J 

I / m>i>j>l ' 7 ^ J ^ 

" / d & 

Co = -mn + ^U-A-{n-m))xi—-2x^ — 

1=1 ^ * 



x^ + x^j f d d 
— X? \ dx-j ^ dxi 

m>i>j>l « J ^ J * 



-2 E 



1=1 



^. .^xf-x^i\^'dxi ^^'dx-j 

Proposition 7. T/ie operator acts on the (unnormalised) orthog- 
onal system {i^k} cls the Jacobi operator 



fc+i) 
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where 

(64) Ak = 4A;^ + (4(m - 2) + 2(n - m))A;3 

+ ((m^ - 4m + 4) + (n - m)(m - 2))A;^, 

= -2A;2 



1 

4' 



Proof. It follows from the above lemma that the operator is a Jacobi 
operator. In order to identify the coefficients A^, B^, and Ck, we eval- 
uate the polynomials at points {xi, 0) := (xi, 0, . . . , 0). Then we have 



= (2 + 8A; + 2A;(2A;- l))(2A;)!xf+' 

4fc^ + 6fc + 2 
= (2A; + 2)(2A: + 1) ^^^^^^^^'°^^ 
= '0fe+i((a;i,O)), 

whence — \. 

We now investigate the action of the operators — that occur 

in >C_ and in >Co. For i and j fixed, we write the symmetric polynomial 
■ipk as a sum (suppressing here the indices k, i and j in order to increase 
readability) 

^i^k^ Yl PcA^){xrxf+xi'xf), 

c>d>0 

where the Pc,d are symmetric polynomials in the variables other than 
Xi and Xj. The operator then acts on the second factor of each term, 
and 

r^._d „ d 

'dx, ^9xj , 2c 2d , 2d 2c^ 

= 2(c - d)ix,x,f''{xf''-''-'^ + ■■■ + xf-'-'^). 



BRANCHING LAWS FOR MINIMAL HOLOMORPHIC REPRESENTATIONS 21 



Evaluating the right hand side at (xi.O) (whence Xi — 0) yields zero 
unless d = 0, in which case we get 2cxf^^ ^\ Therefore, 

= J]pe,o((xi,0))(2ca;f-^))((xi,0)). 

c=0 

We now consider two separate cases. 

(1) If J = 1, then evaluating the polynomial Pc,o at a point (xi, 0) 
yields zero unless it is a constant polynomial, i.e., unless c — k. 
In this case, p^^ — (2/c)!. 

(2) If j 7^ 1, then evaluating Pc,o2ca;^*^'^ at (xi, 0) gives zero unless 
c — 1, in which case we get the value 

2pi,o(xi, 0) = 2 ((^^Yyi) ' ^^^^ " 

= 4A;2(2(A;-l))bf-2. 
Hence, we have 

m>i>j>l ^ J \ * J/ 

= (m - l)2A;(2A;)!xf -2 + ^ ~ ^ ^ 4A;2(2(A; - "2. 

From this, we conclude that 

£_^fc((xi,0)) 

= 2 (^(m - l)2A;(2A;)!xf-2 + " ^ ^ 4A;2(2(A; - l))lxf-^^ xf-^ 

+ {2k{2k - l){2k)\ + {m- 1)4A;2(2(A; - 1))!) 
+ {{n - m)2k{2k)\ + 4(m - l)(n - m)k'^{2{k - 1))!) 
+(4(m2 - 4m + 4) + 4(n - m)(m - 2))(2(/c - l))!a;f "^ 
and hence 

Ak = 4fc^ + (4(m-2) + 2(n-m))P 

+ ((m^ - 4m + 4) + (n - m)(m - 2))A;^. 
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Similarly, we see that 

£o^fc((a;i,0)) = (-mn + (-(n - m) - 4)2A; - 4A;(2A; - l))(2fc)!xf 
-2(m-l)2A;(2/c)!xf 
= (-8P + (-4(m - 1) - 2{n - m) - 4)A; - mn)'ipk{{xi, 0)), 

and hence the value of Bk. □ 

Theorem 8. The Hilhert space M'l" is isometrically isomorphic to the 
Hilbert space L^(S,yu), where 

^ / N r/1 n — m ,,,, 1 n — m 

S = (0, oo) U - + A;)|A; G N, - - + k<0}, 

and /i 25 the measure defined by 

1 T{a + ix)T{b + ix)T{c + ix) " 



(65) / fdfi 2^ 



f{x)dx + 



T{2ix) 

Tja + c)r(c + &)r(& - c)r(a - c) ^ >^ {2c)j{c+l)j{c + b)j{c + a)j ^ ■ 
r(-2c) (c)^.(c-6 + l)^.(c-a + l)/ ^ 

c+j<0 

x/(-(c+jr), 

where the constants a, b, and c are given by 

, , m — 1 n — m 

(66) a = — — + 



6 



2 4 
1 n — m 



2 4 ' 
1 n — m 



2 4 

Under the isomorphism, the operator corresponds to the multipli- 
cation operator / H-i> — (a^ + x^)/. 

Proof. We recall the continuous dual Hahn polynomials, Sk{x^; a, b, c), 
(cf. [16]) defined by 

Sk{x'^;a,b,c) _ f -k,a + ix,a-ix , 
^ ' {a + b)k{a + c)k ' 'V a + b,a + c ' 
Here, (■)fc denotes the Pochhammer symbol defined as 
{t)o = 1, 

{t)k = t{t + l)---{t + k-l), keN+. 
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Suppressing the parameters and denoting the left hand side above by 
S'fc(x^), these polynomials satisfy the recurrence relation 

(68) - (a^ + x^)~Su{x^) = A'^~Sk^^{x^) + B',Su{x^) + C'^Sk+iix"), 
where the recursion constants A'^, _B^, and are given by 

(69) v4'fc = k{k + h + c-l), 

(70) Cfc = {k + a + h){k + a + c), 

(71) B', = -{Ai + C',). 
Under a renormalisation of the form 

S'fc(x^ a, b, c) akSk{x^, a, b, c) := Sk{x'^, a, b, c)", 

where ak is some sequence of complex numbers, the corresponding 
polynomials 5'"^ will also satisfy a recurrence relation of the type in 
( l68ll . with constants, A'^,B'^,C^, given by 





Oik 






B^ 


= Bl 




_ ttfc 







(72) = 
(73) 

(74) Ct = 

From this we can see that the product A'^^j^^C'j^ = A^j^^C^ is invariant. 

Consider now the continuous dual Hahn polynomials with Ski^x"^] a, c), 
with the parameters a, 6, c from (l66l l. These polynomials satisfy the 
orthogonality relation (cf. [16]) 

2 

5'A;(a;^; a, 6, c)5'i(x^; a, 6, c)da; 



1 

2^ 



r(a + ix)r(6 + ix)r(c + ix) 



T{2ix) 

r(a + c)r(c + 6)r(& - c)r(a - c) 

^ V- (2c),(c+l),(c + &),(c + a), . 
^ (c),(c-6 + l),(c-a + l)/ ^ 

c+j<0 

X S'fe(-(c + if] a, 6, c)S'/(-(c + if] a, 6, c) 
(75) = V{k + a + b)T{k + a + c)T{k + b + c)k\6ki. 

By a straightforward computation one sees that the corresponding 
constants A'^,!?^, and are related to the Jacobi constants Ak,Bk, 
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and Ck in ( l65i l by 

We can thus use (l74l ) to define a sequence recursively in such a way 
that the resulting polynomials 5^ satisfy the recurrence relation 

(76)- (a2 + x^)St{x^) = Ak~SU{x^) + Bk~St{x^) + ^^^."^^(x^) 

with the same Jacobi constants as the operator 4£^. More precisely, 
we set 

1 / Tfl\ / Tl\ ~^ 

(78) := 4 (^+2) 

Then = (r(f )r(f ))"^^^4^ (f )^ (f)^, and hence, by ([75]), we have 

(79) fStWh = 4^'(^0^(y)^Q 

(80) = 11^.11?. 

Therefore, the operator Tq : J^/" L'^(T,,dfi) defined by 

(81) % = 

is a unitary operator which diagonalises the restriction of the operator 
£1 to J^/-. □ 

Theorem 9. For each a; G S, there exists a Hilbert space =^ and an 
irreducible unitary spherical representation, tTx, of H on J^x such that 

(1) If Vx G ,y^x is the canonical spherical vector, then there is an 
isometric embedding of Hilbert spaces L^(T,,fi) C j^M'xdn^x) 
given by 

f ^ Sf, 

where Sf{x) := f{x)vx. 

(2) The operator Tq extends uniquely to an H -intertwining unitary 
operator 

(82) T : (vTi, JTi) (^j nxdfi{x),j J^xdfi{x)^ . 
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Proof. The Banach algebra (H) equipped with convolution as mul- 
tiplication carries the structure of a Banach *-algebra when the in- 
volution * is defined as f*{h) = f{h~^). The representation tti of H 
induces a representation of L^{H) by 

(83) 7r(/) = / fih)mih)dh. 

JH 

If V-{H)* denotes the subalgebra of left and right L-invariant L^- 
functions, the closed C*-algebra generated by 7ri(L^(iJ)*) and the 
identity operator is a commutative C*-algebra. Moreover, the Casimir 
operator vri(C) commutes with all the operators vri(/) for / G L^(if)*. 
Hence, (by [l],Vol. I, Thm 1, p. 77), the diagonalisation of the Casimir 
operator yields a simultaneous diagonalisation of the whole commuta- 
tive algebra 7ri(L^(if)*). 

For / e L^(if)*, we let the function / : S C be the multiplier 
corresponding to the operator T7ri(/)T~^ : L^(S,/i) L^(S,/i). For 
each a; e S, we let A^; denote the multiplicative functional 

(84) A.(/) := /(x), 

which clearly is bounded almost everywhere with respect to /i. The 
equality 

(7ri(/)v2o,V5o)i = y A^(/)rf/i(x) 

holds for / G L^(iJ)*, i.e., the positive functional 

(85) $o(/) := K(/)'^o,^o)i,/ e L\H)* 

is expressed as an integral of characters. 

By [28] (Thm. 10) there exists a direct integral decomposition into 
unitary spherical irreducible representations of the form (l82|l , and it 
expresses the functional $o as an integral of characters against the 
corresponding measure. This measure is supported on the characters 
given by positive definite spherical functions. By [25] (Thm. 11.32), 
such an integral expression for bounded positive functionals is unique, 
and hence every character A^; can be expressed by a positive definite 
spherical function 0^^^ as 

A.(/) = / f{h)Mh)dh. 

JH 

The rest now follows from the proof of Thm. 10 in [28] . 
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□ 

5. A SUBREPRESENTATION OF 71i\h 

Recall that the boundary is the disjoint union of m G-orbits. 
More specifically, for j = 1, . . . ,m, let Cj denote the n x m matrix 
with 1 at position (j, j) and all other entries zero. Then 

m 

d^=[j G(ei + ■ ■ ■ + e,) 

and the inclusion 

G(ei + ■ ■ ■ + er+i) C ^(ei + ■ ■ ■ + e^) 

holds for r = 1, . . . , m — 1. The Shilov boundary is the G-orbit of the 
rank m partial isometry Ci + ■ ■ ■ + 6^- It is also the i^'-orbit of this 
element. We consider now the "real part", Y, of the Shilov boundary, 
i.e., 

(86) F:=5nM„„(R). 

Then Y is the homogeneous space H/Pq, where Pq is the maximal 
parabolic subgroup defined by the one dimensional subalgebra 

ao = + ■ ■ ■ + E„) 

of a (cf. (l28l)). We let Pq = MqAqNo be the Langlands decomposition. 
Then Y can also be described as a homogeneous space Y = L/LPiMq. 
Consider the one dimensional representation with character 

(87) /H^|detAd^[^^,(/)| 

of L n Mq. The induced representation Ind^p^^^d det Adj^J-|j^|) is re- 
alised on the space of sections of the density bundle of F = L/LCiMq. 
The representation (l87ll is in fact trivial, and this allows us to define 
an L-invariant section, u, by 

(88) cu(/(L n Mo)) := /e(LnAfo)^o, 

where uq E ViTe(LnMo)) is arbitrary, where V{Te(LnMo)) denotes 
the vector space of densities on Te(LnMo)- The section u then corre- 
sponds to a constant function : L ^ C In the usual way, we 
will sometimes identify u with the measure it defines by integration 
against continuous functions. We then use measure theoretic notation 
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and write Jyipdu for Jyipu. Moreover, we choose c^o in f l88l l so that 
this measure is normalised. 

Using the identification l/[ fl mo — f)/po, the representation (1871) 
extends to the representation 60 of po given by 

(89) So{moaono) = | det(Ad(,/po(moaono)"^|. 

Clearly, 5o('^o«o^o) = 6^''°'^^°^''"'', where po denotes the half sum of the 
restricted roots. The action of H as pullbacks (actually, the inverse 
mapping composed with pullback) on densities is equivalent to the left 
action defined by the representation lndp^{6o). For the extension of 
the function F^j to a Pq equivariant function H C (which we still 
denote by F^), we then have 

(90) F^ikomoaouo) = e-^"'^'"^'"'^^ F^{ko) = e-'^'^'''^^''^ F^{e). 
From this, it follows that 

(91) h*uj{l{L n Mo) = e-2^o(^°s^»('^')oj(/(L n Mq)). 

The action of on F can either be described on the coset space H/ Pq 
in terms of the Langlands decomposition for Po, or in terms of the 
geometric action on the boundary of ^ defined by the Harish-Chandra 
decomposition. The next proposition expresses the transformation of 
(jj under H in terms of the latter description. 

Lemma 10. The density u transforms under the action of H as 

(92) h*uj{v) = Jhiv)(^'>uj{v). 

Proof. The idea of the proof is to use the (non-unique) factorisation 
H = LMqAqNq of H. We prove that the group Nq fixes the reference 
point Ci + ■ ■ ■ + Cm and acts with Jacobian equal to one on the tangent 
space at ei + ■ ■ ■ + Cm, and the group elements in Mo have Jacobian 
equal to one at ei + ■ ■ ■ + Cm- By the chain rule for differentiation, it 
then suffices to prove the statement for all group elements in Aq. 

In the Langlands decomposition pj^jj^ = m © a © n for the minimal 
parabolic subgroup, the subalgebra n is generated by the restricted 
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root spaces 

m>j>i>l 



e » 

m>j>i>l 



©^ 



E* 




u is upper triang. 



where the matrices are written in blocks in such a way that the block- 
rows are of height m, n—m, and m respectively, and the block-columns 
are of width 171,11 — m, and m respectively. 

In the Langlands decomposition trio ® ao ® tIq, the centraliser, mo of 
ao is the direct sum 



mo = m © [) 

m>j>i>l 



E* — E* 5 
3 J 



and 

m 

(93) no= ^+^.©0^. 

m>j>i>l j=l 



The matrices Xq and X^ commute, so in order to prove that the ele- 
ments in A'^o have Jacobian equal to one at ei -|- • • • -|- e^, it suffices to 
consider elements of the form 

/ 1-g q \ 
expXg = 1 , 

V -Q 1 + q J 



exp Xz — 




separately. 
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We have 

exp Xq exp (ei H h e^) = 











) : 


1 












A B 

If we write this matrix in the block form [ ^ 1 , then the K'^- 



C D 

component in the Harish-Chandra decomposition is given by 

A - BD-^C 
D 



'■n+mi 



and hence 

(94) Jexpx,(ei + --- + e^) = 1. 

Next, we consider the action of exp X^. We have 



1-4^ 1 



exp Xz exp(ei + • • • + e^) = | —z 10 
Here, the ir*^-component is given by 



Z*Z t 1 

2 ^ -1- 



/ 1 

ii:^(expX^exp(ei H hej)= 1 

\ 1 

The complex differential of exp at ei + • • • + is then the linear 
mapping 

(95) dexpX,(ei + --- + ejy = ^ \ ^Y, 

where we have identified the tangent spaces with p"*" = M„^(C). 
Clearly, the determinant of this mapping is 

(96) \ = 

Consider now the subgroup Mq. Its Lie algebra mo is reductive with 
Cartan involution given by the restriction of 9 and the corresponding 
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decomposition is 

mo = mo n [ © mo n q. 

The abelian subalgebra a is included in mo fl q, and therefore (cf. [TO] . 
Prop. 7.29) 

(97) monq= |J Ad(/)a. 

We now investigate the Jacobians of arbitrary group elements in A. 

/ A(cosht) Af sinht ) 
expH= i 1 

y Af sinht ) A(cosht) 

where A( cosht ) denotes the m x m diagonal matrix with entries 
coshti, . . . , cosh tm, and the other blocks are analogously defined. Then 

exp(tiEi H h tmEm) exp(ei H h e^) 

A(cosht) A ( cosh t + sinht ) 

1 
A(sinht) A ( cosh t + sinht ) 

The K'^-component is 

K'^{exp(tiEi H h tmEm) exp(ei H h e^)) 

A(ell) 1 

1 

A(el) 

so the differential d{exp(tiEi + ■ ■ ■ + tmEm)){ei + ■ ■ ■ + e^) is the 
mapping 

Yi\ ( A(ez!!)yi 

Y2 l2A(el!) 

where Yi is the upper mxm block of the nxm matrix in the tangent 
space. Counting the multiplicities of the eigenvalues e~*^ , we see that 

(98) Jexp{*,i^,+...+*„i.,„)(ei + ... + em) = e'^^+'^^Sr^^*.. 

If we write a as the orthogonal sum a = ao © (do)"'" (with respect to 
the Killing form), then (oo)"*" consists of those tiEi + ■ ■ ■ + tmEm in a 
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for which EJi^j = 0. From the identities ((91), ([96D, (I97D, and ([98]) 
we can thus conclude that 

(99) Jh{ei H h Cm) = JAo{h){ei H h e^)- 

On the other hand, by (l93l) . 

(100) 2po(t(^i + ■ ■ ■ + ^m)))) 
m(m — 11 



t + m{n — m)t = m{n — l)t, 

z 

(101) e"" 



2 
so 

-2po(t(Si+--+S™)) 



(^exp{t{£;i+-..+i<;„))(ei H V e„,))"+" 



□ 



In what follows, we will define a Hilbert space of functions on the 
manifold Y . Hilbert spaces of a similar kind were also considered by 
Neretin and Olshanski in [19]. One difference is that their spaces were 
not defined using a limit procedure (see the next definition below). 

We begin by introducing some notation. For a continuous function, 
/, on Y and r G (0, 1), we define the function : F — > C by 

(102) F^(m) := y /(t;)det(J„ -rat;*)-^rfcj(f). 

We construct the Hilbert space by requiring that the following space 
of functions be dense. 

Definition 11. Let denote the set of all continuous functions 
/ : F ^ C such that the limit function 

F{u) := limF^(M) 

r— >1 

exists in the supremum norm. 

On ^0 we define a sesquilinear form ( , )'ro by 

(103) {f,gy,, = J^fiu)GiI^du;{u). 
By the Dominated Convergence Theorem, we have 



(104) / f{u)G{u)duj{u) 



lim / f{u) / g{v) det {In — ruv ) duj{v)duj{u), 
r^l 1^^ Jy 
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and hence the form ( , )<^(, is positive semidefinite. Let M denote 
the space of functions of norm zero, i.e., 

(105) Ar={/G^o|(/,/k = 0}. 

Then the quotient space ^o/A/" together with the induced sesquilinear 

form, ( , )<^(j, is a pre-Hilbert space. We define ^ to be the Hilbert 

space completion of with respect to ( , )<i^o. We denote the inner 
product on ^ by ( , )<^. 

Proposition 12. The action t of H on given by 

(106) r(/i)/(r/):= A-i(r^)^/(/i-ir^), 

where (3 = descends to a unitary representation of H on 

Proof. It suffices to prove that the dense subspace ^o/AA of ^ is H- 
invariant and that the action is unitary on ^o/A/". For this, it clearly 
suffices to prove that the space is -ff-invariant, and that H preserves 
the sesquilinear form ( , )<^q, since then the subspace J\f is also H- 
invariant. 

Consider first the mapping / i— >■ F in Definition [TTl We write Ki 
for the reproducing kernel. For h G H, we then have 

J T{h)f{v)K,{ru,v)dLu{v) = J Jhih-'vy^f{h'')Ki{rui,v)dLu{v) 

= JH{vT^^^f{v')K,{rui, hv')duj{v'), 

by the transformation property for the measure uj. By the transfor- 
mation rule for the reproducing kernel Ki, we have 

j Jh{v')-^^^ f{v')Ki{rui, hv')duo{v') 

Jhih^^ruY"^ f{v')Ki{h~^ru, v')duj{v'). 
Therefore, 

lim / T{h)f{v)Ki{ru,v)du{v) = Jh{h~^u)~'^F{h~^u), 
where the convergence is uniform in u, so is if-invariant. 
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Next, take f,g e %. Then, {T{h)f,T{h)g)<^^ is given by 
(t(/i)/,t(/i)^)^„ = J^Jh{h-'u)fih-'u)Jh{h-'u)-^G{h-H)duj{u) 

= / f{v')G(^du;{v') 

JY 

where the second equality follows from the transformation property of 
u. □ 

The next proposition gives a sufficient condition for the Hilbert 
space ^ to be nonzero. 

Proposition 13. The (equivalence class modulo N of the) constant 
function 1 belongs to the pre-Hilbert space ^^jM if and only ifn — m > 
2. 

Proof. Recall that the reproducing kernel has a series expansion 

oo 

det(/„ — zw*)~^ — k\Kk{z, w), 

k=0 

where Kk{z,w) is the reproducing kernel with respect to the Fock- 
Fischer norm for the i^T-type indexed by k. The functions 

K2k{z,v)duj 

are then L-invariant vectors in the K-type 2k and hence differ from 
the L-invariants -0^ by some constants depending on k. We determine 
these by computing the integrals for a suitable choice of z. 
Before we begin with the computations, consider the fibration 

p:Y^S^-\p{v)^v{e^). 

For u e S'^~^ , the fibre p~^{u) can be identified with the set of all rank 
m — 1 partial isometrics from R'" to (R-u)-*". Moreover, p is equivariant 
with respect to the actions of 0(n) on Y and S^~^. Hence the equality 

(107) / fda^ f fopdu, 

JS"^-^ JY 

where a denotes the normalised rotation invariant measure on 5"""^, 
holds for all / G C{S''-^). 
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Choose now z = Aei, where < A < 1. Since zv^ is a matrix of 
rank one, det(/„ — zv^)~^ = (1 — tr(2f*))~^ Hence 



Y 



{l-iY{zv^))-^duj = / {I - XvuY^du = / {I - \p{v)i)-^duj. 
By ffTOTl) . we have 

(I - Xp{v)i)-^du = I {I - \ui)-^da{u). 



Y 



Moreover, 



/ (1 - \ui)-^da{u) = V A^' / n?^da{u). 

The integrands on the right hand side depend only on the first coor- 
dinate, and hence the integrals can be written as integrals over the 
open interval (-1, 1) in M (cf. [24] 1.4.4.). In fact, 

n,da{u) - r(i/2)r((n- l)/2) J ' 
This integral is zero for odd j, and for j = 2k, we have 

'2k +1 n-l\ T{^)T{^) 



r(^) 



Jjl - x^r-^y^-^x^dx = B 
Therefore, 

„ oo ( n\-n ( 2k±l\ 

From this, it follows that for an arbitrary 2; G we have the expansion 



(108) / det{In- zv^)-^dLj{v 



Y 

00 p/nN-p 1^ 2^+1 



^J{^T{^)T{2k+lf'^ 

Since the functions are L-invariant, they are constant on the set 
{ru\u G F, < r < 1}. This value equals 

(109) Mru) = r^'M^) = r'H'^kl (^)^ . 
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Suppose now that |r - r'\ < e. By ( fTOSl ) and ( fTTlQl ). 



Ki{ru, v)duj{v) 



Kiij'u, v)duj{v) 

Y 

r(f)r(^) 



rri)r 



•2A;+n 



-4'= A;! 



and hence we have the estimate 



(110) 



K\{ru,v)dijj{v) — / Ki{r'u,v)duj{v 



Y 



Y 

oo 



< 



E 



r(f)r(^; 



rfi)r 



T{2k 



I) \2J, 



Applying Sterling's formula to the kth term on the right hand side 
yields 



fllll 



r(f)r(^; 



;i)r(2^)r(2fc + 



—A'^klf-) =o(k- 
1) \2)k ^ 



2 



Hence, the sum in (IllOp converges if and only if n—m > 2. In this case, 
the corresponding net {/y i^i(r-, f )(iu;}^ is Cauchy in the supremum 
norm, and hence converges uniformly. □ 

Lemma 14. Consider the representation r in (1106^ . On the space 
of continuous functions on Y, it is equivalent to the representation 



Indpil 



.p\^L w \iX + p) 1), where P is the minimal parabolic subgroup 

is 



defined by the maximal abelian subspace o C p, and A G (o 
defined as 

(112) 
(113) 



- (^^ + P)lao 
-(^A+p)|„x 



2(n-2) 



n 



1 



-Po, 



0. 



In fact, when the continuous functions on Y are identified with right 
L n Mo-invariant functions on L, we can extend them to functions on 
H in such a way that the two representations are equal in this setting. 

Proof. By ( llOip , we can rewrite the action of H in p06l) as 
(114) r{h)f{x) = e-^^°('°sAo(9-ix)j(^(^-i^))^ 
where 

g-'^x = K{g''^x)mo{g~^x)Ao{g''^x)no{g'^x) E LMqAqNq. 
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We now let A G (o*^)* be defined by the requirements pi2p and pi3l) . 

By pi3p . —{iX + p) has to annihilate all the restricted root spaces 
1)e*-e*, and hence be of the form c{El + ■ ■ ■ + E^) for some constant 
c. By (fTnnll it follows that c = -m{n - 2). 

Consider now the parabolically induced representation 

Indp (1 (g) exp{iX + p) O 1) 

acting on on continuous functions on H. By definition, this repre- 
sentation is defined on the space of continuous functions f : H ^ C 
having the P-equivariant property 

(115) fixman) = e-^(^+'')(^°s")/(a;). 
The action of H is given by 

(116) / ^ e-('^+'')(^('^"'"))/(«^(/i"'a;)). 

On the other hand, the restriction of the representation r to the 
space of continuous functions on Y coincides with the if-action defined 
by the parabolically induced representation Indpg(exp). Since P C Pq, 
and 

(^IIY^ g-(iA+p){logA(x)) ^ g-(iA+p){logAo(x))^ 

it follows that 

(118) r{h)f{x) = e-(^^+'')('°s^('^"'"»/(/€(/i-^x), 

where / is the extension of a continuous function on F to a Pq- 
equivariant function on H. This finishes the proof. □ 

Proposition 15. The operator T : ^ 0{^) defined by 



Tf{z) = 1^ f{v) det(4 - zv'y'dcuiv) 



is H-equivariant. 
Proof. We have 

T(r(/i)/)(z) = / .Mh~'v)^f{h-'v)K^iz,v)dcu 
Jh{sf^^f{s)K^{z, hs)duj 



Y 



Y 



Jh{h ^z) / Jh{sf'^"+'^ "+™/(s)K'i(/i ^z,s)di 
ni{h){Tf){z). 



BRANCHING LAWS FOR MINIMAL HOLOMORPHIC REPRESENTATIONS 37 



□ 

Corollary 16. The function Tl is a joint eigenfunction for all oper- 
ators 'Ki{Z), Z G Z{U{[f-)). In particular, it is an eigenfunction for 
the Casimir operator, t^i{C), with eigenvalue _ "^(^~^) _ 

Proof. By Lemma [HI we can identify the extension of constant func- 
tion 1 on F to a function on H with the Harish-Chandra e- function 
ex : H ^ C given by 



(119) ex{h) = e 



-i{\+p){logA{h) 



Moreover, the representation Indp (l®exp(zA+p)(8>l) has infinitesimal 
character iX + p (cf. [101, Ch. VIII). The value of the Casimir element 
is -(a + p)(C) = -((A, A) + (p,p)) = -^ (cf. [llj, Ch. V). □ 

Proposition 17. The function {Tl){z) = Jy det{In — zv^)^^duj{v) 
belongs to 

Proof. We rewrite the series expansion in (|108l) using the orthonormal 
basis {v?fc}, i-e., 

» oo 

(120) / Kx{z,v)duj{v) = y^^ak(pk{z), 

where ak = r(i)r(^''^r(2Ti i) ~- Sterling's formula 

(121) al = ©(A;-^"-'")/^)^ 

and hence the series J2k ^1 converges if and only if — m > 2. □ 

The operator T maps the if-span (the set of all finite linear com- 
binations Cir(/ii)l + ■ ■ ■ + CNT{hi)l, hi E H,Ci E C) of the function 1 
into J^i. We introduce the temporary notation if ■ 1 to denote this 
subspace. Moreover, we let Mh-i := Af (1 H ■ 1. 

Proposition 18. The equality 

(122) {Tf,Tf), = {fJ)^, 
holds for f e H ■ 1. 
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Proof. For / e if • 1 and r e (0, 1), consider the function Tf{r-). We 
have 

(123) Tfirz) = J^f{v)K,{rz,v)du;{v) 

(124) = J f{v)Ki{z,rv)duj{v). 
The square of the J^-norm is then given by 

\\Tp{r-)\\i = J^J^f{OmMrC,r7])duj{C)duj{7^). 

These norms are uniformly bounded in r, and hence there is a conver- 
gent sequence {T/(rfc-)}fc with respect to the ^-norm. Since point 
evaluation functionals are continuous, we also have pointwise conver- 
gence, and hence this limit function is Tf. Therefore, 

WTfWl = lim ||T/(r,.)||? 

fc— >oo 

= hm / / f{C)J(^K,{r\,ri)du;{Odu;{ri) 
"•-^^ Jy Jy 

= ll/lllo- 

□ 

We let Ti denote the restriction of the operator T to the subspace 
H ■ 1. Then, we have the following corollary. 

Corollary 19. For the operator Ti : H ■ 1 ^ Mi, 

(125) ker Ti = A/W.i- 

The operator Ti then descends to an operator Ui : H ■ 1/Mh-i — 
J^i. Now let Ti denote the Hilbert space completion of the space 
H ■ 1/Mh-i- We keep the letter r to denote the representation of H 
of this space (in reality, the representation we mean is derived from 
T by first restricting, then descending to a quotient, and, finally, by 
extending uniquely to a Hilbert space completion). 

Proposition 20. The representation t of H onTi is irreducible. 

Proof. The representation r is i?-cyclic with a spherical (L-invariant) 
vector. Hence, there exists a unitary, if-equivariant direct integral 
decomposition 

(126) S-.n^ I HxdniX), 

J A 
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where A is a subset of the bounded spherical functions (or rather, the 
functionals on a that parametrise them), /i is some measure on A, and 
Tlx is the canonical spherical unitary representation corresponding to 
the spherical function 0a. For each A, we let v\ denote the canonical 
spherical vector in Tix. 

Suppose now that r is not irreducible, i.e., the set A is not a single- 
ton set. Then, we can choose two disjoint open subsets ^1,^2 of A. 
We define vectors si and S2 in the Hilbert space H\dfi by 



^i(A) ^ 



v\, if A G fii 

Oa, otherwise 

■ua, ifAefi2 

Oa, otherwise 



The vectors S^^si and S~^S2 are then linearly independent spherical 
vectors in H. But, clearly, the only spherical vectors in Ti are the 
(cosets modulo Mh-i of the) constant functions; a contradiction. □ 

We are now ready to state a subrepresentation theorem. The proof 
follows from Prop. [HI the above corollary, and Cor. [T6l 

Theorem 21. The operator Ui can be extended to an isometric H- 
intertwining operator 

(127) U -.n^j^i. 

Its image is isomorphic to the spherical unitary representation corre- 
sponding to the discrete point {i (| — ^^-^)} in the spectral decompo- 
sition for the Casimir operator r^iiC). 
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